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SENSE  MAKING:  BIASES  AND  SUPPORT  SOLUTIONS 


Analysts  base  their  hypotheses  concerning  terrorist  groups  and  terrorist  attacks  generally  on  large 
amounts  of  information ,  which  may  be  uncertain ,  inconsistent  and/or  incomplete.  The  present  paper 
addresses  the  questions  of  how  an  analyst  constructs  a  causal  scenario,  which  biases  occur  and  how  this 
intelligence  process  can  be  supported.  In  the  first  part  a  model  is  presented  that  describes  the  subsequent 
steps  in  the  analysis  process.  Several  biases  may  negatively  affect  the  quality  of  the  analysis  process. 
These  are  described  in  the  second  part  of  the  paper.  Subsequently,  an  experiment  is  described  in  which  we 
manipulated  the  presentation  of  information:  in  a  time  line,  a  relation  scheme  or  unstructured.  In  general, 
the  results  show  that  people  generate  fewer  motives  in  the  unstructured  task  condition  and  that 
expectations  increase  probability  assessment  of  related  hypotheses.  The  last  part  of  the  paper  presents 
support  concepts  which  aid  analysts  to  critically  think  about  their  own  analysis:  To  what  extent  is  a 
hypothesis  anchored  in  the  data  at  hand,  how  to  explain  data  not  in  line  with  the  hypothesis  and  which 
other  hypotheses  would  be  possible  as  well?  Supporting  analysts  in  critical  thinking  not  only  improves 
situation  awareness  and  decision  making,  but  allows  for  better  communication  with  others  as  well. 


1.0  INTRODUCTION 

Intelligence  services  are  more  often  confronted  with  mistakes  such  as  the  use  of  unsubstantial  evidence, 
unrecognised  threat  and  poor  exchange  of  information.  Some  of  these  problems  can  be  solved  by  technical 
solutions,  such  information  exchange  or  integration  of  information,  but  for  others  human  information 
processing  needs  to  be  supported.  This  not  only  involves  the  availability  of  information  but  also  the 
quality  of  the  analysis.  The  fist  part  of  the  paper  addresses  the  way  information  is  analysed.  In  the  second 
part  of  the  paper  several  biases  are  described.  The  third  part  describes  a  recent  experiment  on  the  effect  of 
visualisation  on  sense-making  and  the  fourth  part  proposes  some  ways  to  support  intelligence  analysis. 

1.1  The  analysis  process 

Recently,  Pirolli  &  Card  (2005)  presented  a  scheme  that  describes  subsequent  steps  in  an  intelligence 
process  (see  Figure  1).  Basically,  two  different  loops  can  be  distinguished:  a  foraging  loop  aimed  at  the 
gathering,  selecting  and  evaluation  of  information  and  a  sense  making  loop  aimed  at  finding  the  meaning 
of  the  information. 
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Figure  1:  model  of  intelligence  process  (Pirolli  &  Card,  2005) 

The  squares  represent  how  data  is  transformed  and  the  circles  indicate  the  various  steps  in  the  process. 
Data  undergo  the  following  transformations:  the  external  data  sources  (left  in  Figure  1)  are  the  raw  data, 
mainly  text.  From  these  data  sources,  information  is  selected  that  is  relevant  for  further  processing 
(shoebox).  The  evidence  file  contains  pieces  of  evidence  that  have  been  selected  from  the  shoebox.  These 
pieces  of  evidence  are  subsequently  visualised  in  some  kind  of  scheme,  which  can  take  on  different  forms, 
such  as  a  time  line  or  a  relation  scheme.  On  the  basis  of  this  scheme,  interpretations  can  be  made  in  order 
to  build  up  a  scenario.  Hypotheses  are  tentative  conclusions,  founded  by  arguments.  Ultimately  a  kind  of 
presentation  will  be  given  to  answer  the  question  posed  to  the  analyst. 

Two  kinds  of  processes  can  be  distinguished:  bottom-up  and  top-down  (indicated  by  the  circles  in  Figure 
1).  The  bottom-up  processes  move  from  left  to  right,  and  include  operations  like  searching  and  filtering  of 
information,  extraction  and  schematization.  Top-down  processes,  on  the  other  hand,  move  from  right  to 
left  and  are  mainly  used  to  find  support  for  assumptions  or  conclusions.  A  new  hypothesis,  for  example, 
may  give  reason  to  go  back  to  the  raw  data  in  order  to  check  whether  the  hypothesis  at  hand  fits  with  the 
data. 

1.2  Hypotheses  and  scenarios 

Analysts  who  are  trying  to  make  sense  of  a  set  of  information,  strive  for  a  story  or  scenario  that  is  logical, 
consistent  and  plausible  (Pennington  &  Hastie,  1993).  The  resulting  scenario  is  partly  based  on  the  facts 
available,  but  inevitably  interpretations  and  inferences  are  made  as  well.  These  interpretations  and 
inferences  are  based  on  general  world  knowledge  and  domain-specific  knowledge.  Pennington  and  Hastie 
(1993)  showed  that,  due  to  this  mechanism,  jurors  can  infer  totally  different  verdicts  on  the  basis  of  the 
very  same  information.  As  personal  beliefs  affect  the  interpretation  of  information  and  the  way  missing 
data  is  filled  in,  jurors  come  up  with  different  stories  and,  as  a  consequence,  different  verdicts. 
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The  human  information  processing  system  has  limited  capacity:  working  memory  can  only  manage  one 
hypothesis  at  a  time.  Research  has  indicated  that  persons  generate  only  a  small  proportion  of  scenarios 
possible,  when  they  are  required  to  make  sense  of  a  set  of  data  (Dougherty,  et  al.,  1997).  One  starts  with  a 
limited  set  of  possible  scenarios,  which  is  reduced  as  soon  as  possible,  by  means  of  consistency  checks.  If 
a  particular  hypothesis  is  generated,  its  consistency  is  checked  against  the  data  available,  and  if  it  seems 
unlikely,  it  is  eliminated  from  the  set. 

A  complete  scenario  is  difficult  to  test,  and  for  that  reason  it  is  fractioned  into  smaller  units,  the 
hypotheses.  These  hypotheses  mostly  concern  central  statements  in  the  scenario,  which  can  be  checked 
against  the  data  available.  The  probability  of  a  hypothesis  is  determined  by  comparing  it  to  alternative 
hypotheses.  It  has  been  shown  that  individuals  are  more  realistic  about  the  likelihood  of  a  hypothesis  when 
they  are  forced  to  think  of  an  alternative  hypothesis.  Thinking  about  the  underpinning  of  a  particular 
hypothesis,  inadvertently  leads  to  tunnel  vision.  Considering  alternative  hypotheses  breaks  through  this 
inertia  (Koehler,  1991). 


2.0  EXPERIENCE  AND  BIASES  IN  INTELLIGENCE  ANALYSIS 

As  mentioned  above,  a  scenario  results  from  an  interpretation  of  data  available.  As  such,  it  contains 
subjective  elements  resulting  from  previous  experience  with  similar  cases,  general  world  knowledge  and 
individual  beliefs.  In  the  present  chapter,  I  will  discuss  the  influence  of  experience  and  some  biases  on 
intelligence  analysis. 

2.1  Influence  of  experience 

Due  to  years  of  experience,  experts  have  built  up  patterns  that  represent  the  most  important  features  of  the 
task.  Even  though  it  has  been  suggested  that  expertise  also  depends  on  personal  abilities,  it  has  been 
shown  that  expertise  is  mainly  defined  by  these  domain-specific  schemes  (Ericsson  &  Lehmann,  1996). 
The  content  of  these  patterns  depends  on  the  specific  domain  in  which  the  expertise  had  been  built  up. 
Physicians,  for  example,  built  up  scripts  that  represent  the  relations  between  symptoms,  context  and 
disease  (Boshuizen  &  Schmidt,  1992),  whereas  market  intelligence  analysts  have  developed  schemes 
representing  characteristics  of  players,  developments  and  opportunities  (Pirolli  &  Card,  2005). 

Schemes  enable  a  quick  structuring  of  huge  amounts  of  information,  in  order  to  make  sense  of  the 
situation.  This  doesn’t  mean  that  it  is  passive  pattern  recognition.  In  intelligence  analysis  there  is  a 
continuous  interplay  between  data  available  and  knowledge  already  stored  in  memory:  data  may  trigger 
knowledge,  whereas  knowledge  directs  the  way  information  is  gathered  and  interpreted. 

The  extent  to  which  experience  positively  contributes  to  the  intelligence  process,  is  dependent  on  the 
learning  environment  (Hogarth,  2001).  An  important  mechanism  that  causes  incorrect  knowledge  to  be 
built  up  is  that  persons  almost  exclusively  learn  from  what  they  see,  and  ignore  content  that  cannot  be 
seen.  In  intelligence  analysis,  for  example,  the  focus  is  often  on  evidence  that  provides  an  underpinning  of 
the  hypothesis  at  hand,  whereas  data  that  might  falsify  the  hypothesis  is  ignored.  Naturally,  individuals  do 
not  tend  to  falsify  their  hypothesis,  but  base  their  knowledge  on  positive  observations.  An  implication  of 
this  tendency  is  that  analysts  should  be  supported  in  testing  their  hypothesis  critically. 

As  far  as  hypothesis  generation  is  concerned,  Weber,  et  al.  (1993)  showed  that  experts  more  often  related 
a  hypothesis  to  own  experience.  In  our  own  experiment  with  crime  analysts  we  found  no  relationship 
between  years  of  experience  and  the  analysis  process  (Kerstholt  &  Eikelboom,  submitted).  Even  though  it 
was  not  systematically  investigated,  the  results  suggested  that  specific  experiences  with  a  case  triggered  a 
particular  hypothesis.  Part  of  the  analysts,  for  example,  mentioned  a  Tover  boy’  scenario  as  an  explanation 
for  the  disappearance  of  a  young  woman  and  mostly  referred  to  their  own  experience  with  a  similar  case. 
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In  all,  our  results  suggested  that  years  of  experience  did  not  affect  the  analysis  process,  only  domain 
specific  knowledge. 

2.2  Biases  and  expectations 

Knowledge  or  beliefs  create  a  context  against  which  new  evidence  is  interpreted.  An  illustration  of  this 
effect  is  shown  in  Figure  2. 


12 

A  13  C 
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Figure  2:  Illustration  of  the  context  effect 


The  symbol  in  the  middle  can  either  be  seen  as  a  number  (13)  or  as  a  letter  (B).  The  symbol  is  somewhat 
ambiguous  and  this  ambiguity  is  solved  by  taking  the  context  into  account.  If  the  series  is  read  from  top  to 
bottom,  the  symbol  is  interpreted  as  13,  but  if  it  is  read  from  left  to  right  the  symbol  is  interpreted  as  a  ‘B’ 
(Risinger,  et  al.,  2002).  Context  therefore  induces  an  expectation  with  regard  to  the  interpretation  of  an 
ambiguous  stimulus  with  the  result  that  ones  sees  what  one  expects  to  see. 

Particularly  in  juror  decision  making  numerous  studies  have  investigated  the  effect  of  extra-legal  factors, 
such  as  the  defendant’s  status  (Bornstein,  1994),  race  (Foley,  Adams  &  Goodson,  1996)  or  pretrial 
publicity  (Otto,  Penrod  &  Dexter,  1994).  As  noted  by  Bornstein  (1994),  many  extra-legal  factors  exert 
their  effect  by  drawing  on  juror’s  prior  beliefs.  A  juror  might,  for  example,  belief  that  a  defendant  with  a 
particular  background  or  race  is  more  likely  to  commit  a  crime.  Often  this  belief  will  not  have  any 
diagnostic  value  and  it  will  therefore  affect  the  judgment  negatively.  An  extra-legal  factor  could  also  work 
as  a  scheme  or  framework  to  which  new  evidence  is  related.  An  opening  statement,  for  example,  creates 
such  a  schema  (Spiecker  &  Worthington  (2003),  but  also  a  hypothesis  that  is  tentatively  favored  as  the 
trial  progresses.  Carlson  and  Russo  (2001)  showed  that  when  a  juror  has  such  a  tentatively  favored 
hypothesis,  new  evidence  is  somewhat  distorted  in  order  to  fit  the  hypothesis. 

In  an  experiment  with  crime  analysts  an  expectation  was  induced  by  presenting  analysts  a  particular 
interpretation  of  the  case,  before  they  started  with  their  analysis  (Kerstholt  &  Eikelboom,  submitted). 
Analysts  who  had  seen  the  prior  interpretation  considered  the  suggested  scenario  more  likely  than  analysts 
who  had  not  received  it.  This  result  held  for  both  inexperienced  and  experienced  analysts.  As  analysts  are 
generally  well  aware  of  potential  biases  due  to  prior  beliefs,  it  can  be  concluded  that  awareness  alone  is 
not  sufficient  to  eliminate  biased  interpretation. 


3.0  INFORMATION  PRESENTATION 

Support  for  intelligence  analysis  is  mostly  geared  towards  filtering  and  structuring  huge  amounts  of 
information.  By  means  of  visualisation  of  relationships  between  various  data,  patterns  can  be  detected 
more  easily  which  reduces  workload  of  the  analyst  (Xiang,  et  al.,  2005).  However,  data  can  be  visualised 
in  different  ways,  for  example  by  a  timeline  or  relation  scheme,  which  might  differentially  affect  the  sense 
making  process  of  the  analyst. 
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In  order  to  investigate  whether  this  is  the  case,  we  have  conducted  an  experiment  in  which  we  presented 
participants  with  two  visualisations  of  the  same  data:  a  timeline  and  a  relation  scheme  and  we  compared 
both  visualisations  with  an  unstructured  way  of  information  presentation.  We  expected  that  participants 
would  generate  more  hypotheses  in  the  unstructured  information  presentation  condition,  as  there  would  be 
no  directing  effect  of  the  visualisation. 

3.1  Design  of  the  experiment 

Forty-five  participants  (students)  were  required  to  generate  explanations  for  the  disappearance  of  a  young 
woman.  The  woman  came  from  a  very  religious  family  and  it  was  known  that  her  father  was  opposed  to 
her  relationship  with  a  foreign  man.  Participants  received  25  witness  statements  and  were  required  to 
generate  as  many  explanations  for  the  disappearance  as  possible. 

The  witness  statements  could  be  presented  in  three  different  ways:  1)  by  means  of  a  time  line,  meaning 
that  all  events  and  related  witness  statements  were  sorted  by  time;  2)  relation  scheme,  showing  all  persons 
involved  and  their  mutual  relationship;  3)  unstructured,  meaning  that  all  witness  statements  were 
presented  in  a  random  order.  The  different  visualisations  were  presented  on  a  computer  screen  and  by 
clicking  on  a  related  button  the  underlying  witness  statement  could  be  requested.  In  order  to  compel 
participants  to  think  about  the  most  relevant  information,  they  could  only  request  80%  of  the  statements 
available. 

In  addition  to  visualisation  we  also  manipulated  prior  expectations.  Half  of  the  participants  received  an 
interpretation  of  the  case  before  they  did  the  task,  while  the  other  half  did  not  have  this  interpretation.  This 
interpretation  suggested  that  the  father  had  played  an  active  role  in  the  disappearance  because  he  resented 
his  daughter’s  relationship  with  a  foreign  man.  We  registered  what  information  was  requested,  the  number 
and  kind  of  hypotheses  generated,  and  probability  assessments. 

3.2  Results 

Two  different  aspects  could  be  distinguished  in  the  hypotheses  generated:  the  person  involved  and  a 
motive.  A  possibility  was,  for  example,  that  the  woman  had  left  on  her  own  account,  and  motives  could  be 
that  she  wanted  to  escape  from  her  father’s  pressure,  that  she  was  abused  by  her  boy  friend,  that  she  was 
pregnant  etc.  Participants  generated  on  average  4.9  (sd=2.0)  persons  or  events  that  could  be  responsible 
for  the  disappearance  of  the  woman  and  5.7  motives  (sd=2.3).  There  was  a  marginally  significant 
interaction  between  presentation  mode  and  kind  of  response,  that  is,  persons  or  motives  (F(2,32)=3.18, 
p=.06,  see  Figure  3).  Univariate  analysis  showed  that  the  number  of  motives  differed  across  presentation 
conditions  (F(2,32)=3,56,  p<.05)  and  not  the  number  of  persons  or  events  (F92,32)=1.22),  p>.l).  In  the 
unstructured  task  condition,  significantly  fewer  motives  were  mentioned  than  in  the  relation  scheme  and 
time  line  condition.  As  far  as  number  of  different  persons  was  concerned,  no  differences  were  found. 
There  was  no  effect  of  expectation  (F1,32)<1). 
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Figure  3:  Number  of  hypotheses  generated  in  all  experimental  conditions 


Participants  in  the  expectation  condition  considered  the  ‘father  scenario’  more  probable  than  participants 
who  had  not  received  a  prior  interpretation  (.39  versus  .57,  F(l,41)=6.43,  p<.05).  The  friend-scenario  was 
marginally  significant  across  expectation  conditions  (F(l,41)=3.60,  p=.06).  In  the  expectation  condition, 
the  possibility  that  her  boy  friend  played  a  central  role  in  the  disappearance  of  the  woman  was  considered 
higher  than  in  the  non-expectation  condition  (.51  versus  .40).  The  self-scenario  (she  left  on  own  account) 
was  considered  equally  likely  in  both  expectation  condition  (F(1,41)<1).  Presentation  mode  did  not  affect 
probability  assessments. 

3.3  Discussion 

We  predicted  that  more  hypotheses  would  be  generated  in  the  unstructured  condition  as  there  would  be  no 
influence  of  data  visualisation  into  a  particular  direction.  This  prediction  was  not  supported  by  the  data. 
We  made  a  distinction  between  the  number  of  different  persons  or  events  that  could  be  responsible  for  the 
disappearance  and  the  motives  that  were  mentioned.  In  the  unstructured  task  condition  significantly  fewer 
motives  were  mentioned  than  in  the  time  line  and  relation  scheme  condition. 

A  possible  explanation  for  this  effect  is  the  increased  workload  in  the  unstructured  task  condition,  as  a 
representation  needed  to  be  built  up  from  scratch.  In  the  time  line  and  relation  scheme  condition  a 
representation  was  given,  and  participants  could  solely  focus  on  hypothesis  generation.  In  these 
conditions,  participants  generated  significantly  more  motives  than  persons,  whereas  participants  in  the 
unstructured  task  condition  generated  approximately  one  motive  per  person  or  event.  In  the  unstructured 
task  condition  participants  were  therefore  less  creative  in  generating  possible  explanations  than  in  the 
other  task  conditions. 

Expectations  did  not  affect  hypothesis  generation,  but  only  probability  assessment  of  the  hypothesis  that 
was  suggested  in  the  prior  interpretation.  Participants  who  had  received  the  prior  interpretation  gave  a 
higher  probability  assessment  to  this  hypothesis  than  participants  who  had  not  received  a  prior 
interpretation.  This  effect  was  independent  of  the  way  the  information  was  presented. 
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The  only  unexpected  effect  was  the  marginally  significant  interaction  between  probability  assessment  of 
the  ‘friend-scenario’  and  expectation.  The  prior  interpretation  only  concerned  the  father,  but  the  friend- 
scenario  was  considered  more  likely  in  the  expectation  condition,  as  well.  In  the  expectation  manipulation 
it  was  suggested  that  the  father  had  done  something  to  his  daughter  because  he  could  not  accept  her 
relationship  with  a  foreign  man.  Possibly,  participants  thought  of  more  extreme  motives  after  hearing  this 
interpretation  which  also  increased  the  probability  of  the  ‘friend’  scenario. 

To  summarise,  participants  seem  to  be  less  creative  in  generating  possible  reasons  when  information  is 
presented  unstructured.  Prior  interpretations  do,  on  the  other  hand,  not  affect  the  number  of  explanations 
generated.  The  second  conclusion  is  that  expectations  have  an  effect  on  probability  assessment.  A 
hypothesis  that  agrees  with  a  prior  interpretation  receives  a  higher  probability  score. 


4.0  SUPPORT  SOLUTIONS 

One  of  the  main  threats  to  biased  information  processing  concerns  the  influence  of  prior  beliefs.  Due  to 
these  beliefs  information  is  selectively  gathered,  it  is  interpreted  in  the  direction  of  the  belief,  and 
information  may  even  be  distorted  in  light  of  a  currently  held  belief.  Such  a  set  of  beliefs  has  also  been 
called  a  mindset  and  as  indicated  by  Heuer  (1999)  they  are  neither  bad  nor  good.  They  are  simply 
unavoidable.  It  is  not  possible  to  process  a  large  amount  of  information  without  knowledge  of  what  to 
expect,  what  is  important  and  how  things  are  related.  Over  and  above,  even  if  analysts  would  be  aware  of 
these  mind-sets,  it  would  presumably  not  eliminate  flawed  judgements.  In  order  to  achieve  a  more 
objective  assessment,  essential  assumptions  underlying  a  causal  scenario  need  to  be  made  explicit,  in  order 
to  allow  for  critical  tests.  In  other  words,  what  is  needed  is  that  analysts  are  supported  in  meta-cognitive 
strategies  that  enable  them  to  critically  evaluate  their  own  assessment  (Cohen  et  al.,  1996;  Hogarth,  2001). 

As  far  as  the  generation  of  hypotheses  is  considered,  it  is  recommendable  to  always  think  of  an  alternative 
hypothesis,  as  this  will  make  the  assessment  of  the  main  hypothesis  more  realistic  (Koehler,  1991).  An 
analyst  can  be  supported  in  generating  alternative  hypotheses  by  showing  patterns  in  the  data  that  may 
point  to  a  particular  hypothesis.  Support  could  be  provided  by  providing  quantitative  data  such  as 
frequency  of  contact  between  persons  involved,  or  by  allowing  search  possibilities  such  as  common 
ground  between  two  persons  (they  may  share,  for  example,  the  same  garage  or  go  to  the  same  bar). 
Searching  for  patterns  and  relationships  in  data  is  a  topic  in  which  computers  clearly  outperform  humans, 
as  computational  load  is  high. 

In  testing  a  hypothesis,  research  indicates  that  people  tend  to  only  look  for  confirming  evidence  and  are 
quite  reluctant  to  switch  to  an  alternative  hypothesis.  In  this  phase  of  the  analysis  process  people  should  be 
supported  in  critically  relate  the  hypothesis  stated  to  the  data  at  hand.  First,  it  could  be  visualised  how  the 
hypothesis  is  anchored  in  the  data:  which  data  support  the  hypothesis  and  which  data  don’t?  Second, 
people  could  be  made  aware  of  unknown  parts  of  the  scenario.  One  of  the  scenarios  we  used  in  our  study 
on  crime  analysis,  for  example,  concerned  trade  in  women.  Information  was  provided  about  persons  and 
property,  but  not  about  the  financial  flow.  A  semantic  network  indicating  all  aspects  involved  in  trade  in 
women  could  have  made  this  lack  of  information  visible,  allowing  the  analyst  to  be  aware  of  missing 
information.  Third,  analysts  could  be  made  aware  of  critical  assumptions  underlying  their  hypothesis  by 
some  kind  of  sensitivity  analysis:  if  this  information  was  not  true,  how  would  it  affect  the  likelihood  of  my 
hypothesis?  This  kind  of  support  would  make  analysts  aware  of  critical  information.  This  critical 
information  needs  to  be  reliable  and  as  such  the  tool  allows  for  a  differentiation  between  relevant  and  less 
relevant  information  in  the  top-down  check  for  information  reliability.  Essentially,  analysts  need  support 
in  making  a  more  explicit  underpinning  of  the  hypothesis  generated.  This  will  not  only  result  in  better 
judgments,  but  allows  for  better  communication  and  evaluation  by  others  as  well. 
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